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The problem of deciding when a stationary stochastic process is a function of a finite Markov 
process has been considered by several authors, leading to an elegant necessary and sufficient condi- 
tion. Taking a different approach, this note uses elementary ring theory to prove that a certain explicitly 
constructed stationary ergodic process is not a function of any finite Markov process. 
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1. Introduction 

The problem of deciding when a stationary stochastic process is a function of a finite Markov 
process has been considered by several authors (fl, 2, 4-6]), l leading to an elegant necessary 
and sufficient condition (Heller, [6J). This condition involves both (a) the finite dimensionality of 
a certain module over the real numbers, and (b) the requirement that a certain cone in this module 
be polyhedral (an idea used earlier by Dharmadhikari [2]). The output symbols of the stochastic 
process act as linear operators on the above-mentioned module. 

Taking a different approach, this note uses elementary ring theory to prove that a certain 
specific stationary ergodic process is not a function of any finite Markov process. The structure 
of this process — although not the idea — resembles that of an example of Fox and Rubin [4]. 

Section 2 presents the principal tool for the verification, namely a sufficient condition for a 
"source" (in the language of information theory) not to be a function of any finite Markov process. 
This condition and its verification are purely algebraic. 

Section 3 gives the explicit description of the example, plus the verification of its properties. 
A related conjecture is posed in section 4. 

The balance of the present section 1 consists of definitions and other background material 
provided to make the exposition more nearly self-contained. 

Let A be a fixed finite nonempty set; the sources to be considered emit "symbols" from this 
"alphabet" A as outputs. LetA™ be the set of all doubly infinite sequences x={#f}~ x of members 
of A; each such sequence can be considered a possible time history of the source's outputs, with 
symbol xt emitted at time t. The shift operator T:A X ^> A* is defined by 

Tx= y where y t = x t+ \. (1.1) 

A cylinder set is a subset of A* of the form 

C{J, g) = {x€A*>:x t = g(t) for all tej} 
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where J is some finite subset of the integers (positive, negative, or zero) and g:J—> A is some func- 
tion. If J= {1, 2, . . ., k} for some k, then C(7, g) is called an elementary cylinder set. It follows 
from the finiteness of A that every cylinder set is a finite union of elementary cylinder sets. 

Let 2 be the Borel field generated by all cylinder sets (or equivalently, all elementary cylinder 
sets). We now define an information source over A to be a probability measure /jl on (^4°°, 2). Such 
a source will be denoted [A, /jl]. In terms of the notation in (1.1), the source is called stationary if 
p(TS)=p(S) for every measurable subset S of ^°°, and is called ergodic \{TS = S (where S is 
measureable) implies p{S) =0 or p(S) = 1. 

Let / n ={l,2,. . .,rc}, and suppose there is a function /:/ n — » A , and an n-state Markov 
chain with transition-matrix entries p(i,j) and stationary probabilities p(i), such that, for every 
elementary cylinder set 

C=C({1,2,. . .,k},g), 
we have 

p(C)=Xp(i 1 )p(i u i2) • • . p(ik-uik) (1.2) 

where the sum is over all sequences {h, io, • • . , ik] over I n such that f(i t ) =g(t) for t= 1, 2, . . . ,/r. 
Note that this entirely determines p, so that [A, p] is specified by the finite set of data {p(i), 
p(i, j)} together with/. In this case, [A, p] is called a function of a (finite) Markov process; it is 
automatically stationary, and is ergodic if the Markov chain is. 

If / n , in the last paragraph, is replaced by the set of all natural numbers, we are led instead 
to the concept of "function of a denumerable-state Markov process." The example to be given 
later falls into this category, but not into the previous one, as will be proven. 

2. An Algebraic Condition 

Let Lly denote the set of all polynomials in TV variables with integer coefficients. For the in- 
formation source [A , p] , define the following subring of the ring of real numbers: 

V(A,fi) = {p:p = P(n(C 1 ), . . .,^(C\ V )) for some TV >0, some PeU N , 

and some cylinder sets Ci,C2, . . ., Cjv}. 

Suppose in particular that [A, pu] is a function of an n-state Markov chain with parameters {p(i), 
p(i, j)}. Then it follows .by (1.2) that the measure of any elementary cylinder set is the value, at 
(p(l), . . ., p(n), p(l, 1), . . ., p(n, n)) of a polynomial in n + n 2 variables with integer coeffi- 
cients. It follows that the same is true of the measure of any cylinder set, and so is true of every 
member of V(A , p) . That is, if we set 

W N {u u . . ., u N ) ={p :p = P(u l , . . ., u N ) for some PeU^}, 
then we have, 

V(A,ii) cr n+n2 (p(l),. . .,p(n),p(l,l),. . .<p(n,n)) 

whenever [A , p] is a function of the n-state Markov chain with the indicated parameters. 

This observation provides the basis for the following result. 

THEOREM 1: Suppose there are real numbers {pi} 7 ! 1 such that the ring generated over the in- 
tegers by V(A, p) U {pi} f 1 includes all the rationals. Then the source [A, p] is not a function of 
any finite Markov process. 

This will be shown to follow from the next theorem, whose conclusion, in order to support an 
inductive proof, is stronger than is needed for Theorem 1. 
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THEOREM 2: Let U= (ui, . . ., u N ) be a fixed N-tuple of real numbers, k a fixed positive 
integer, (Di, . . ., D k ) fixed polynomials in I1 N with each Dj(U) 4= 0. Then the set R N = {p:p = P(U)/ 
II f [Dj(U)] a(j) /or some Pell \ oft^ so/ne integral a(j) ^ 0} does no£ include all the rationals. 

We first show that Theorem 2 implies Theorem 1. In Theorem 2 let N=m + n-\- n 2 ,let U={p\, . . ., 
p w , p(l), . . ., p{n), p(l, 1), . . ., p(n, n)), and let all Z)j = 1. It follows from (2.1) that, if 
[A, p] is a function of the Markov process with the indicated parameters {p(i), p(i\ j)}, then 
V(A, pi) C R\. Since pieRx, and R\ is a ring over the integers, the ring R * generated over the 
integers by V(A, /x) U {p/}/" also lies in R\. By Theorem 2, /? * cannot contain all rationals, and 
so Theorem 1 is proved. It is interesting that the proof uses only (1.2), and not the further properties 

p(i,i)^0, p(»)*0, 

2jP^'-/') = 1 ' ^iP(i)p(i>J)=P(J)- 

We turn now to the proof of Theorem 2, based on a suggestion by M. Newman. The nota- 
tion R[zi, . . . ,z n ] will mean the ring of polynomials in the indeterminates Z\ , . . . ,z n with 
coefficients in a ring R. The proof is by induction on N. 

Case 1 : N = 1. Suppose the conclusion is false, i.e., that there is U= u x such that R \ includes 
all the rationals. Let / denote the ring of integers. 

U=u x must be algebraic over the rationals, since for any prime p, 1/peftj, which implies an 
equation of the form 

n*[D j (c/)]«o>-pP(f/)=o 



in which not all coefficients can vanish if p is chosen so as not to divide all coefficients of the 
polynomial fl^Dj.Now let Uj=u i \ ) and let u { f (2^i^m) be the ith conjugate of U\ over the 
rationals, so that 1 1 / [ l (z — u^) is the minimal polynomial of U\ over the rationals. Then for a suit- 
able integer \ , 

g(z)=\Il^(z-u^)e\\u 

Let {en} f denote the elementary symmetric functions of m variables, and put 

s i = (Ti(u i \ ) . . . , df), 

so that 

g(z) = k(z m -s l z m - 1 + . . ,+ (-l) m s m ). 

It follows that all ks,€l. 

Let p be a prime, to be determined later. As above, l/peR\ leads to an equation f(ui) = 
where 

/-riH^)" (J) -pp 

for some appropriate PeH\ and nonnegative integers a (J). Since g is a constant multiple of the 
minimal polynomial of U\, f (ui)=0 implies that /= gh for some polynomial h with rational co- 
efficients. Since g(u { f ) =0 for all i, all/(a ( J ) ) =0 and so 
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i/p=p(^>)/nf[^(^l ) )] a(j) , 

implying 

Hpm = Il™P (d$ )/U i T 1 II,*, [Dj (df ) ] «0). (2.2) 

In the above note that 

cannot be zero. For otherwise some Dj(u ( ]>)=0, implying Dj(ui)=0, contrary to hypothesis. 
Now consider the symmetric polynomials 

p*(z u . . .,z m )=n?P(zt), 

Df{z u • • .,z m ) = nfZ> j (z i ). 
By [9; p. 97], there are polynomials 7T, 8je/[zi, . . ., z m ] such that 

P*(Z1, . . .,Z m )= 77[cri(zi, . . .,Z m ), . . .,GT m (zi, . . . , Z m ) ] , 

D*(z U . . .,Z m )= :r 8j[o-i(zi, . . .,2m), . . .,0- m (zi, . . .,Z m )]. 

It follows from (2.2) that 

l/p*=ir(*i, . . .,ta)/n}[%(ji,. . .,5 m )]«0>. (2.3) 

Let cfO) = deg [8J, and choose integer 6 > so that 

d=X*d(j)a(j) + b^deg[ir]; 

since all Xs,e/, this implies that X d 7r(si, . . . , s tn ) is an integers. Note also that X d(j) S/(si, . . ., 5 m ) 
is an integer ^- independent of p. 
Now, (2.3) yields 

Hp=p m - 1 wl\ b U.^(w j ) a ^ 

Choosing p to be a prime which divides neither X nor any Wj, we obtain a contradiction. (Recall 
that X does not depend on p.) 

Case 2: TV > 1. Again we assume that Rn contains all the rationals. As in the proof for Case 1, 
U= (tti, . . ., iiy) cannot be algebraically independent over the rationals. Without loss of gener- 
ality, we assume U\ is algebraically dependent on U = (u u . . ., ^v-i). Expand the notation to 
read 

R, = R N (U;D U . . .,/),). 

We will show that there exist D u . . ., D k +ieYl N -i with a\\Dj(U) =1=0 such thatR N -i(U; D u .. .,D k+i ) 
contains all the rationals. This yields the desired contradiction of the induction hypothesis. 

Let Rn be the ring generated over the integers by U, i.e., 

Ro = {p:p = P(U) for some PeUv-i}, 
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and letFo be the quotient field of Rq. Then u^is algebraic overF () ; \etu^ )= u N and let u ( $(2^i^m) 
be the conjugates of u y over Fo, so that 11™ (z — u$) is the minimal polynomial of u v overF . Then 
for a suitable Xe/?o, 

«r(*)=xn«(i-i^)€RoW. 

Define {en}™ as above, and si = (Ti(d*\ . . ., ^ ( v m) ); then all KsteRu. 

As in Case 1, we consider a prime p. The requirement 1/peRx leads as before first to an 
equation f(U) =0 where 

/-n{(/)j)«o)-pp, 

for some appropriate Pell a and nonnegative integers a(j). This can be rewritten/* ( a, v ) = where 

f* (z)= II* [Dj(U,z)]«W- P P(U,z). 
Arguing as above we find that all/*(tt ( $ ) = 0, implying 

Up =P(U,u[{))IU>i[Dj(U,u[i))]«^ 
and thus 

Hp m = UfP(U, a^J/IIiS, n^, [£;(£/, a$ ) ]«<» . (2.4) 

Now consider the symmetric polynomials 

P*(Z U . • • ,Zm) = n?P([7, Z|)€/?o[zi, . . . ,Zm], 
Z>f(Zi, . . . 9 Z m ) = nfDj(U,Zi)€R [zu • • • ,^m]. 
As above, there are polynomials 77, 3j eR [zi , . . . , z m ] such that (2.4) yields 

L/p m =ir(si f • • • 9 s m )IU\[8j(su • • ■ . s»»)] a0) ; (2>5) 

as before A, the 5/ and the 8, are independent of p. Defining d(j), b, and ^ as above, we have 

l/p=[p m " 1 X%(5i, . . . ,«m)]/X»n*(uy)«U) (2.6) 

where Wj=k d ^8j(si, . . . , 5 w )e/?o. We may denote the numerator of (2.6), which lies in /?o, 
as P(U). Similarly let Wj = Dj(U) for some Djell/v_i, let k= D k+i (U) where Z)fc+i€ll;v-i, and 
leta(A+l) = 6. Then (2.6) yields 

Hp = P(U)IU k t 1 [D j (U)] a ^eR N . 1 (U;D 1 , . . . ,D, +1 ). 

Thus if R\(U; D±, . . . , /)/,) contains all rationals, or equivalently all prime-reciprocals 1/p, then 
so does R\-\{U; Di, . . . , D*+i). 
This completes the proof. 

3. The Construction 

Our example is constructed as a function of a denumerable Markov chain M, which will be 
described next. The state-space of M consists of a special state So, and an infinite lower-triangular 
array of states Sy; the i\h row of the array consists of the states S*i, S*2» ■ • •* S;;. The probability 
of transition from So to a row-leading state Sn is </,-, where the g/'s are positive numbers (to be speci- 
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fied) with V } qi= 1; no other states are directly accessible from So. The other direct transitions are 
from Sn to Si 2 , Si2 to Si3, • • -, and from Su back to So, each with probability 1. 
We now exhibit a vector a, with positive components summing to 1, such that 

aP = a (3.1) 

where P is the transition matrix of M. The entries of a will be denoted p (So) and p(Stj), with obvious 
interpretation. For some x > 0, set 

p(So)=x, p(Sij) = qiX. (3.2) 

The equations of (3.1) which correspond respectively to So, to any Sa, and to anySi,j+i are 

Zjip(Su) - l=p(So), 

p(So)qi = p(Su), 

p(Sij) • l=p(St,j+i), 

and all are satisfied; hence (3.1) holds. It only remains to satisfy the normalization 

p(So)-&j^iP(Sij) = l, 

which identifies x as 

x=[l^iiqi]-K (3.3) 

The qt will be chosen so that the above makes sense, i.e., 

^iiqi < a). (3.4) 

Take the alphabet^ = {0, 1}. If X denotes the state-space of M, define f:%—> A by 

/(S,)=0, f(Sij) = l. 

[A, jll] is now defined as the function of M determined by /. 
Consider the elementary cylinder set 

Ci = C({l,2, . . .,H-2},*«) 

where gi{\) =gi(i + 2) =0 and g"/=l otherwise. Then /x(C/) is the probability of the source pro- 
ducing a pair of O's separated by exactly i l's. The sum (1.2) here collapses to a single term, yielding 

fi(Ci) = qiX. 

Thus if {qi} can be selected to include, for each prime p, a rational number with denominator 
divisible by p, then the ring generated over the integers by V(A, /jl) U {l/x} will contain all qi, 
hence all reciprocals of primes, hence all rationals. By Theorem 1, [A, fx] cannot be a function 
of any finite Markov process. 

Let 7Ti be the ith prime (iTi = 1), and set 

q i =l/2 i 7T i (i**2). (3.5) 
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Since 77, s 2 in (3.5), we have 

i = 2 / = 2 

and so (3.4) holds for any finite gi ; since 

£ <//<£ l/2 i+1 = l/4<l, 

I* = 2 j = 2 

there is a unique q\ > such that V qf*=l. Thus the goal described in the previous paragraph 
has now been accomplished. i=l 

Although the main discussion is now complete, certain technical points which were passed 
over above must still be verified. It must be shown that M is ergodic (as a Markov chain), that the 
vector a defined in (3.2) is its unique stationary vector, and that [A, p] is (as promised) ergodic. 

With V (as above) the state-space of A/, the identity map of V induces an infinite-alphabet 

source V, /x* which is a function of M. Let V x be the set of doubly-infinite sequences {crt}^ 

over V , and T* the shift operator on V x . Then any function [A, p] of M , induced by a function 

/; V — > A , is ergodic if V » M* * s ergodic. This follows, in terms of the function/ 00 ^V 00- * A* arising 
from/, from the facts 

f*T* = Tf*; /n(fi) = /H*[(/-)- 1 (fi)] 



for any measurable subset B of A x . 



Thus it suffices to prove that 
t, satisfies 



V , /x* is an ergodic source which, for any Se V and any integer 



/x*{xe4 x :^ = S} = probability of S as given by a. 



(3.6) 



The reader is now referred to the discussion in [8; p. 52| of limit properties of Markov chains; 
we shall use the terminology of that discussion. The chain M is clearly irreducible and nonperiodic. 
By (3.4), the mean recurrence time of So is finite, so that So (and hence all states) are persistent. 
By [8; p. 56], M has a unique invariant instantaneous distribution. Since (3.1) holds, this distribution 
must be a; hence (3.6) holds. In addition, letting P denote the transition matrix of M, we have 
for any a,re^ 



lim n ^ oc (P n ) Ta = a (J . 



(3.7) 



The fact that 



5>* 



is ergodic in the sense used in this paper can now be proved from (3.7) 
by elementary but lengthy arguments; instead we refer the reader to Doob [3; p. 460] for an elegant 
proof using martingales. 

This completes the discussion. 



4. Conjecture 

According to (2.1), if [A, p] is a function of a finite Markov chain, then V(A, p) is imbedded in a 
finately generated ring (of real numbers) over the integers. We conjecture that this necessary 
condition is not sufficient: 

CONJECTURE: There is an information source [A, p] on a finite alphabet A with the following 
two properties: 

(1) V(A,/x) is a finitely generated ring over the integers; 

(2) ,[A, p]is not a function of any finite Markov process. 
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